To be distasteful or obnoxious to someone's sensibilities; regularly displease: I don't quite know why, but that woman rubs me the wrong way
[1883+; rub the hair the wrong way is found by 1868]